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This article is part of a series written for people responsible for making decisions about health policies and 
progranfimes and for those who support these decision makers. 

Policymakers and those supporting them may find themselves in one or more of the following three 
situations that will require them to characterise the costs and consequences of options to address a 
problem. These are: I. A decision has already been taken and their role is to maximise the benefits of an 
option, minimise its harms, optimise the impacts achieved for the money spent, and (if there is substantial 
uncertainty about the likely costs and consequences of the option) to design a monitoring and evaluation 
plan, 2. A policymaking process is already underway and their role is to assess the options presented to 
them, or 3. A policymaking process has not yet begun and their role is therefore to identify options, 
characterise the costs and consequences of these options, and look for windows of opportunity in which 
to act. In situations like these, research evidence, particularly about benefits, harms, and costs, can help to 
inform whether an option can be considered viable. In this article, we suggest six questions that can be used 
to guide those involved in identifying policy and programme options to address a high-priority problem, and 
to characterise the costs and consequences of these options. These are: I . Has an appropriate set of 
options been identified to address a problem? 2. What benefits are important to those who will be affected 
and which benefits are likely to be achieved with each option? 3. What harms are important to those who 
will be affected and which harms are likely to arise with each option? 4. What are the local costs of each 
option and is there local evidence about their cost-effectiveness? 5. What adaptations might be made to 
any given option and could they alter its benefits, harms and costs? 6. Which stakeholder views and 
experiences might influence an option's acceptability and its benefits, harms, and costs? 
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About STP 

This article is part of a series written for people responsible for 
making decisions about health policies and programmes and for 
those who support these decision makers. The series is intended 
to help such people to ensure that their decisions are well- 
informed by the best available research evidence. The SUP- 
PORT tools and the ways in which they might can be used are 
described in more detail in the Introduction to this series [1]. A 
glossary for the entire series is attached to each article (see 
Additional File 1). Links to Spanish, Portuguese, French and 
Chinese translations of this series can be found on the SUP- 
PORT website http://www.support-collaboration.org . Feedback 
about how to improve this tool and others in this series is wel- 
come, and should be sent to: STP@nokc.no. 

Scenarios 

Scenario 1 : You are a senior civil servant and will be submitting 
a brief report to the Minister of Health regarding the evidence 
to support a number of options to address a high-priority prob- 
lem. You are concerned about whether the current draft of the 
report includes a reasonable set of options. You are also con- 
cerned about whether the report addresses the likely (Questions 
about each option that can reasonably be answered through the 
use of research evidence. 

Scenario 2: You work in the Ministry of Health and are prepar- 
ing a brief report about options to address a high-priority prob- 
lem that you have been examining in great depth. All that you 
have been told is that the report should present three options 
and focus only on what the research evidence says about each 
option. 

Scenario 3: You work in an independent unit that supports the 
Ministry of Health in its use of evidence in policymaking. You 
are preparing a detailed research report for the Ministry of 
Health about what is known and not known about options to 
address a high-priority problem. You have been told what 
options to examine but want guidance on the types of research 
evidence about each option that could be used to inform a 
choice between options. 

Background 

For poliq^makers (Scenario 1), this article suggests a 
number of questions that they might ask their staff to con- 
sider when preparing a brief report about options to 
address a problem. For those who support policymakers 
(Scenarios 2 and 3), this article suggests a number of ques- 
tions to guide the identification of options and the char- 
acterisation of each option's costs and consequences. This 
article is the second of three articles about clarifying evi- 
dence needs (see also Articles 4 and 6 [2,3])- Figure 1 out- 
lines the processes involved in clarifying these evidence 
needs. 



Policymakers and those supporting them may find them- 
selves in one or more of the following three situations that 
will require them to characterise the costs and conse- 
quences of options to address a problem. Firstly, a prob- 
lem may already have been framed in a particular way, an 
option selected to address the problem, and a political 
constituency mobilised to support such decisions. (A var- 
iant of this situation is that an option may be selected first 
and then a problem identified later as a motivation for the 
option.) In this circumstance, the best option for those 
who support policymakers is to assist them in identifying 
how to maximise the benefits of the option, how to min- 
imise its harms, and how to optimise the impacts 
achieved for the money spent. In addition, if there is sub- 
stantial uncertainty about the likely costs and conse- 
quences of an option, a monitoring and evaluation plan 
can be designed to ensure that policymakers will have the 
right information to hand at a defined point in the future. 
This will help them to decide whether a policy should be 
left unchanged, modified or repealed (or whether a pro- 
gramme should be continued, modified or discontinued). 
Article 18 in this series describes how to develop a moni- 
toring and evaluation plan [4]. 

Secondly, policymakers may find themselves in a scenario 
in which they are actively engaged in a policymaking proc- 
ess. This may mean that policymakers will need to partic- 
ipate in events in which options are being actively 
debated, meet with 'policy entrepreneurs' who want to 
persuade them to endorse a particular option, and 
respond to feedback about the operation of an existing 
policy or programme [5]. In this second scenario, they will 
need to assess the options presented to them, the degree 
of attention being given to the problem that the option is 
meant to address, and any political events that may 
present a window of opportunity during which particular 
actions could be undertaken. 

Thirdly, policymakers may find themselves in a situation 
where they will have more open, strategic opportunities in 
which they are able to define a problem, to identify 
options, to characterise the costs of the options and their 
consequences, and to look for windows of opportunity to 
undertake preferred actions. Such opportunities are rare 
and calculations about when to act will need to be strate- 
gic. 

A policy or programme can be deemed to be an appropri- 
ate solution if it is technically feasible, fits within domi- 
nant values and the current provincial/national mood, 
and is acceptable in terms of budget workability and the 
likely degree of political support or opposition [5]. 
Research evidence can form part of this mix in several 
ways and help to determine the following details about a 
chosen option: 
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Identifying needs 
for research evidence 



Clarify a problem 



Frame options to address a problem 

(and describe each option's costs 
and consequences) 



Option 1 

• Benefits 

• Harms 

• Costs/cost- 
effectiveness 



Option 2 

• Benefits 

• Harms 

• Costs/cost- 
effectiveness 



Option 3 

• Benefits 

• Harms 

• Costs/cost- 
effectiveness 



Implement an option 



Figure I 

Clarifying evidence needs. 



• Whether it is technically feasible - for example, an 
option's benefits may have been shown to be substantial 
and its harms acceptably low. Alternatively, the key ele- 
ments of the policy or programme may have been shown 
to be consistent with those elements critical to the success 
of the option in other settings 

• Whether it fits with dominant values and the current 
national mood and is acceptable in terms of likely politi- 
cal support or opposition - interviews with stakeholders, 
for example, may reveal whether or not it is perceived to 
be acceptable, and 

• Whether it is acceptable in terms of its budget impact 
(and whether its value for money has been demonstrated) 

Interviews with policymakers have confirmed that they 
place a high value on research evidence about the benefits, 
harms and costs of options [6]. 



Questions to consider 

The following questions can guide how to identify 
options and characterise their costs and consequences: 

1. Has an appropriate set of options been identified to 
address a problem? 

2. What benefits are important to those who will be 
affected and which benefits are likely to be achieved with 
each option? 

3. What harms are important to those who will be affected 
and which harms are likely to arise with each option? 

4. What are the local costs of each option and is there local 
evidence about their cost-effectiveness? 

5. What adaptations might be made to any given option 
and could they alter its benefits, harms and costs? 
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6. Which stakeholder views and experiences might influ- 
ence an option's acceptability and its benefits, harms, and 
costs? 

/. Has an appropriate set of options been identified to 
address a problem? 

Initial work should focus broadly on the options that 
could affiect the problem identified. Creative thinking 
about this topic can be encouraged by identifying options 
that affect either or both of the following: 

• The provision of a cost-effective programme, service or 
drug, and 

• The health system arrangements that determine whether 
cost-effective programmes, services or drugs are provided 
to those who need them 

Policymakers and other stakeholders with clinical back- 
grounds often focus largely on issues related to pro- 
grammes, services or drugs. But at the same time they also 
tend to neglect concerns related to the health system 
arrangements that are needed to ensure a high coverage 
rate for the very same programmes, services or drugs that 
matter both to them and to consumers. These health sys- 
tem arrangements may include: 

• Delivery arrangements: such as who the programme, 

service or drug is targeted at; who it is provided by; where 
the care is provided and what information and communi- 
cation technology is used to provide it; and the safety and 
quality systems used 

• Financial arrangements: such as who finances the rele- 
vant parts of the system, programme, service or drug; how 
organisations are funded to deliver the programme, serv- 
ice or drug; how professionals are remunerated to provide 
these; how patients/consumers are presented with incen- 
tives to use it; and how resources are allocated to it, and 

• Governance arrangements: such as who has the policy, 
organisational, commercial and professional authority 
and accountability for those parts of the health system 
that could play a role in addressing the problem 

A key next step is then a consideration of whether these 
elements can stand alone as options or whether they can 
be bundled together to form new options appropriate to 
specific local contexts. 

Policymakers may be able to identify existing frameworks 
that enable the identification of policy or programme 
options. These frameworks may be the focus of reports in 
their own right. For example, the Chronic Care Model 
provides a framework for considering how contributions 



to effective chronic care can be made through self-man- 
agement support, decision support, delivery system 
design, clinical information systems, the health system, 
and the community more generally [7]. Alternatively, 
frameworks may be embedded in systematic reviews or in 
overviews of systematic reviews if they are used to struc- 
ture the search for, and presentation of, research evidence. 
For example, an overview of systematic reviews provides a 
framework for addressing challenges related to human 
resources for health [8]. One dimension of this framework 
addressed the training, regulatory and financial mecha- 
nisms (i.e. the policy and programme options) that could 
be employed. The second dimension addressed the sup- 
ply, distribution, efficient use and performance of health- 
care providers (i.e. the option's potential consequences). 
However, multiple competing frameworks may exist, and 
there is often no empirical evidence to support the use of 
one framework over another. Moreover, like the options 
they are meant to help to identify, frameworks may not be 
mutually exclusive. 

Table 1 provides an example of how the teams supporting 
the widespread use of artemisinin-based combination 
therapy in Africa identified relevant options and then 
approached the characterisation of their costs and conse- 
quences, using Questions 2 to 6 listed below as prompts. 

2. What benefits are important to those who will be 
affected and which benefits are likely to be achieved with 
each option? 

The second stage in framing options involves characteris- 
ing their costs and consequences. The first step in this sec- 
ond phase is to determine the likely benefits (or positive 
effects) of each option. Policymakers need to decide 
which benefits are likely to be important to those who will 
be affected by the decisions taken. Some of the studies 
consulted, for example, may focus on issues related to sur- 
vival while others may address health-related quality of 
life issues. Still others may focus on 'intermediate' out- 
comes such as coverage rates for an effective treatment. 
Policymakers also need to decide whether they are more 
interested in particular groups of people (e.g. children, 
adults or the elderly) and particular comparisons (e.g. 
comparing the option of doing nothing with the option of 
providing standard care). 

The acronym 'POCO' refers to the four key elements that 
must be considered in order to enable the identification of 

research evidence about the benefits of particular options, 
and to ensure that such evidence is used effectively: 

• People (e.g. elderly patients with multiple chronic con- 
ditions) 

• Option (e.g. case management) 
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Table I: Supporting the widespread use of artemisinin-based combination therapy to treat malaria 

The Evidence-Informed Policy Networks (EVIPNet) in ten sub-Saharan African countries described the costs and consequences of three options 
considered viable in these countries for the support of the widespread use of artemisinin-based combination therapy to treat malaria. The impetus 
for these activities was the 2006 WHO guidelines on malaria treatment which endorsed artemisinin-based combination therapy (ACT) to treat 
uncomplicated falciparum malaria [13]. In order to support the widespread use of ACT, national governments in regions with seasonal or endemic 
malaria had to determine whether to confirm or change: 

• Delivery arrangements: including who should dispense ACT (when, where and how), and who should be involved in surveillance, pharmacovigilance 
and the diagnosis and treatment of atypical cases 

• Financial arrangements for patients (e.g. drug subsidies) and for prescribers (amongst others), and 

• Financial arrangements for patients (e.g. drug subsidies) and for prescribers (amongst others), and 

• Governance arrangements: including which ACT and other anti-malarial drugs should be registered and licensed for sale (i.e. which drugs, the 
dosage regimes, and the packaging required), how they could be marketed, who could prescribe them (and how), who could sell or dispense them 
(and how), and what safeguards should be applied to protect against counterfeit or substandard drugs 

EVIPNet teams from each participating country considered options consisting of different 'bundles' of heath system arrangements. One country, for 
example, considered: 

• Using community health workers for the presumptive treatment of uncomplicated malaria with ACT (a delivery arrangement) 

• Introducing ACT subsidies within the private sector to support their use (a financial arrangement) and regulating adherence to the subsidy policy 
(a governance arrangement), and 

• Providing incentives to prescribers 

(specifically nurses and doctors) for a time-limited period to encourage transition to the new treatment (a financial arrangement) 

The teams, consisting of individuals such as those in involved in the second and third scenarios outlined earlier, then approached the task of 
describing the costs and consequences for each option using Questions 2-6 as prompts. 



Type of information about each option 



Benefits 



Potential harms 



Costs and cost-effectiveness 



Key elements of the option (how and why it works) 



Views and experiences of stakeholders 



Examples of the nature of the research evidence sought about 
each option 



• People: everyone except groups other than children under five years of 
age (who were being treated under a separate programme) and 
pregnant women 

(whose cases of malaria were considered 'complicated' and hence 
beyond the remit of this element of the WHO guideline) 

• Option: see above 

• Comparison: status quo 

• Outcomes: both process indicators (e.g. coverage rates achieved) and 
outcome indicators (e.g. survival) 



• As above except for outcomes where process indicators of interest 
included the adherence of community health workers to non-malaria 
related guidelines. This was because of a fear that ACT would be 
provided at the expense of treating other important conditions 



• Costs collected in their own setting 

• Economic evaluation conducted using a societal viewpoint given that 
policymakers were acting in their role as stewards for the entire health 
system, and not just as payers for publicly financed programmes, 
services and drugs 



■ Policymakers had already invested heavily in community health 
workers and wanted to know whether the shared attributes of 
community health workers and lay health workers were sufficient to 
allow them to expect similar benefits to those achieved only with lay 
health workers [14] 



• Policymakers were aware that a large proportion of malaria treatments 
were dispensed by 'medicine sellers' [15] rather than health 
professionals or lay health workers. They therefore wanted to know 
more about the views and experiences of these sellers 
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Table 2: Types of study designs well suited to providing particular types of information about options 



Type of information about the option Study designs well suited to providing Definition 

the information 



Benefits Randomised controlled trials • Experimental study in which Individuals are 

(I.e. positive effects) randomly allocated to be exposed to different 

policy and programme options (e.g. using the 
toss of a coin or a list of random numbers 
generated by a computer) 



Interrupted time series • Study using observations at multiple time 

points before and after a policy or programme 
is Introduced (this Is referred to as an 
'interruption'). The design attempts to detect 
whether a policy or programme has had an 
effect significantly greater than any underlying 
trend over time 



Controlled before/after studies* • Study in which observations are made before 

and after the implementation of a policy or 
programme, both in a group that is exposed to 
the policy or programme and in a control 
group that is not. Data collection Is done 
concurrently in the two groups 



Potential harms (I.e. negative effects) 



Effectiveness studies (see above) 



Observational studies 



• Study in which observations are made about 
those exposed to a policy or programme. Data 
could be drawn from administrative databases, 
community surveys or other sources 



Costs and cost-effectiveness 



Cost-effectiveness studies 



• Study in which the relative expenditures 
(costs) and outcomes (effects) of two or more 
courses of action are compared 



Key elements of the option 
(how and why it works) 



Qualitative studies carried out alongside a 
study of effects (i.e. process evaluations) 



• Study conducted in natural settings and 
usually aimed at interpreting or making sense of 
phenomena in terms of the meanings people 
bring to them. Typically, narrative data are 
collected from individuals or groups of 
'informants' (through interviews, focus groups, 
participant observation) or from documents. 
These are then interpreted by researchers 



Views and experiences of stakeholders 



Qualitative studies 



• See above 



Observational studies 



• See above 



* These studies can be very time-consuming to find yet provide little Information of value. This Is due to the strong likelihood that those who have 
been exposed to an option, and those who have not been exposed to the option, differ In Important ways. Impacts may be attributable therefore to 
differences between the groups rather than to differences in exposure to a particular option 



• Comparison (e.g. routine care), and 

• Outcome (e.g. health-related quality of life) 

Searches for evidence should be as precise as possible 
about identifying those option features that are most 
important to policymakers and other stakeholders. Poli- 
cymakers should also assess the extent to which the evi- 
dence they find addresses the questions they are asking. 



Those studies best suited to answering questions about 
benefits are randomised controlled trials, interrupted time 
series, and controlled before/after studies. All of these are 
characterised by the care taken to minimise the possibility 
that the measured effect of a policy is attributable to 
another factor which has not been measured (see Table 2 
for an overview of the option information required and 
the associated study types). Very often policymakers will 
be able to find systematic reviews of these types of studies 
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Issue Why it is important to consider the issue Source of additional information 



Quality • Research evidence of low quality (I.e. that is not valid, • Article 8 in this series addresses how to assess the quality of 

credible or rigorous) can give policymakers a false Impression systematic reviews [ 1 6] 

of the likely costs and consequences of an option Article 16 addresses how to use a balance sheet Incorporating 

assessments of the quality of evidence [ 1 7] 



Applicability • Research evidence produced in other jurisdictions can be 
valuable, but policymakers need to consider how likely it is 
that the costs and consequences of an option would be 
different in their setting 



Equity 



and this will save them a significant amount of time. In 
Article 7 in this series, we discuss how to find systematic 
reviews [9]. Once such systematic reviews have been 
found, policymakers then need to assess their quality and 
examine the findings in terms of their local applicability 
and their incorporation of considerations related to 
equity (see Table 3). 

3. What harms are important to those who will be affected 
and which harms are likely with each option? 

In this next step, the likely harms (or negative effects) of 
each option are determined. Again, policymakers will 
need to decide which harms are likely to be important to 
those who will be affected by the decisions they take. 
Some studies may address very infrequent outcomes such 
as death. Others may address frequent outcomes like the 
minor side effects of a drug, or focus on 'intermediate' 
outcomes such as the abandonment of routine tasks by 
lay health workers who have been asked to take on a new 
task. The 'POCO' acronym referred to earlier may also be 
used to structure searches for evidence of harms. 

The types of studies best suited to answering questions 
about harms are more diverse. Information about harms 
can sometimes be derived from effectiveness studies. But 
more frequently, information can be found in observa- 
tional studies that track those 'exposed' to an option, 
whether or not the exposure was part of a particular test of 
the option (e.g. a large-scale drug surveillance system). 
The pros and cons of these different data sources have 
been outlined elsewhere [10]. Policymakers will some- 
times be able to find systematic reviews of these types of 
studies and will need to assess their quality and applica- 
bility, as well as incorporate equity considerations. (See 
Article 7 for further information on finding systematic 
reviews [9].) Local evidence about harms may also be 
found by policymakers and this issue is discussed in Arti- 



• Article 9 in this series addresses how to assess the applicability of 
the findings from systematic reviews to a specific setting [18] 



• Article 10 addresses how to take equity into consideration when 
assessing the findings of a systematic review [19] 



de 1 1 in this series [11]. Once potential harms have been 
identified, the next step is then to identify what, if any, 
mitigating actions can be taken to reduce these harms. 

4. What are the local costs of each option and is there local 
evidence about their cost-effectiveness? 

The next step in characterising the costs and consequences 
of the options is to determine each option's costs and, if 
possible, its relative cost-effectiveness. Two options may 
both be effective but one may deliver better outcomes for 
a given cost, or it may achieve the same outcomes at less 
cost. Article 12 in this series addresses ways in which 
research evidence about resource use and costs can be 
incorporated in the assessment of options [12]. In Article 
12 we discuss how data about costs need to be collected in 
the setting where the options are being considered. And it 
includes a discussion on how research evidence about 
cost-effectiveness is often limited by a lack of rigour in 
estimating effects, as well as by challenges in interpreting 
the valuation of resources being used, and by the 'black 
box' nature of the modelling. 

Economic evaluations can often provide a useful frame- 
work for thinking through issues related to cost-effective- 
ness - even if policymakers need to treat the results of any 
given economic evaluation cautiously, just as they would 
for other tj^jes of studies. Economic evaluations, it should 
be remembered, are always written from a particular per- 
spective, whether it be that of a provider, a payer, or of 
society at large. Policymakers and other stakeholders need 
to be aware of the particular viewpoint they themselves 
adopt for any given economic analysis. 

5. What adaptations might be made to any given option 
and could they alter its benefits, harms and costs? 

The penultimate step in characterising the costs and con- 
sequences of an option is to determine whether there 



• Research evidence focused on overall effects or effects 
among advantaged groups can be valuable. However, 
policymakers need to consider how likely it is that the costs 
and consequences of an option would be different in 
disadvantaged groups 
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Characterising the costs and consequence of options involves finding and using many types of research evidence. When available, systematic 
reviews (the subject of Article 7) can help to characterise the benefits, harms, and key features of the options, as v^ell as the views and 
experiences of stakeholders [9]. In the absence of systematic reviews, single studies must be found. Economic evaluations can help to 
characterise the cost-effectiveness of the options. 

The first set of steps involved in finding such reviews and studies includes: 

• Drawing up a list of words or phrases that capture the option (e.g. replacing 'health professionals who currently prescribe an anti-malarial drug' 
with 'lay health workers'), synonyms for each option 

(e.g. substitution), and alternative spellings for each option (e.g. doctor, doctors, physician, physicians, medical, medicine) 

• Deciding whether systematic reviews or single research studies are the focus of the search, and 

• Providing any additional details that limit the search (e.g. children, adults) 
The second set of steps includes: 

• Choosing those words and phrases that would all need to be present in order for the article to be identified (e.g. substitution, lay heath 
worker, and systematic review), connecting them with 'and', and placing each term in brackets 

• Choosing those words and phrases for which only one would need to be present (e.g. physician and its synonyms), connecting them with 'or', 
and putting each term in brackets, and 

• Connecting sets of brackets using 'and' 
The third set of steps includes: 

• Opening in an Internet browser the relevant database: 

» Program in Policy Decision-making/Canadian Cochrane Network and Centre (PPD/CCNC) database http://www. researchtopol icy.ca/ 
search/reviews.aspx for systematic reviews of studies about health system arrangements (benefits, harms, key features, and stakeholders' 
views and experiences) - see Article 7 for additional information [9] 

" Cochrane Library's Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE) http:// 
thecochranelibrary.com for systematic reviews of programmes, services and drugs (benefits and possibly harms) - see Article 7 for additional 
information [9] 

" Cochrane Library's Economic Evaluation Database (EED) http://thecochranelibrary.com for economic evaluations 

o PubMed http://www.ncbi.nlm.nih.gov/pubmed for the 'hedges' (i.e. validated search strategies) to find specific types of single studies (harms, 
key features, and views and experiences of stakeholders) - see Article 4 for additional information [2] 

• Entering the words and phrases, as well as the Boolean operators 'and'/'or' in the search field, and 

• Clicking the appropriate icon to initiate the search 



might be significant interest in, or pressure, to adapt an 
option that has been tried elsewhere. In this instance, pol- 
icymakers need to search specifically for qualitative stud- 
ies (sometimes called process evaluations) carried out 
alongside studies of effects. Such studies can help to iden- 
tify how and why an option works. These assessments can 
then be used to inform judgements as to whether particu- 
lar elements of an option are critically important (and 
hence need to be retained), and which elements of an 
option are not important (and hence could be either 
dropped or modified). Article 4 in this series provides tips 
for finding qualitative studies [2]. 

6. Which stakeholder views and experiences might 
influence an option's acceptability and its benefits, harms 
and costs? 

The final step in characterising the costs and consequences 
of options is to determine whether the views and experi- 
ences of stakeholders could influence the acceptability 
and impact of the options. Stakeholders may include 
healthcare recipients and citizens, healthcare providers, 
managers working in healthcare organisations, and poli- 
cymakers. If influence is likely, then policymakers and 
those who support them need to seek out qualitative stud- 
ies that specifically examine the views and experiences of 
such stakeholders. (Again, Article 4 in this series outlines 
tips related to finding qualitative studies) [2]. 



Table 4 provides guidance on identifying different tj^es of 
research evidence. 

Conclusion 

An appropriate set of options for a specific local context 
can be identified by combining creative thinking with 
generic taxonomies (like the one used to organise the 
PPD/CCNC database) or frameworks specific to a given 
issue or domain, or else bundling these combinations 
together. Each option can be assessed in terms of its likely 
local benefits, harms and costs or cost-effectiveness, and 
in terms of whether any adaptations might alter these ben- 
efits, harms and costs and related stakeholder views and 
experiences. Policymakers should take into accoimt their 
quality and the local applicability of their findings when 
using systematic reviews to answer questions about bene- 
fits, harms and costs. They should also consider key equity 
considerations. These considerations form the focus of the 
following articles in this series: Article 8 (on assessing the 
quality of a systematic review). Article 9 (on assessing the 
local applicability of the findings of a systematic review) 
and Article 10 (on taking equity into consideration when 
assessing the findings of a systematic review). Policymak- 
ers should also be aware that they will face a practical chal- 
lenge in assessing the relative value of the benefits, harms 
and costs and in making trade-offs between them. This 
topic is the focus of Article 16 which discusses the use of 
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research evidence in balancing the pros and cons of differ- 
ent options. 

Resources 

Useful documents and further reading 

- Kingdon JW: Agendas, Alternatives, and Public Policies, 2 
edn. New York, USA: Longman; 2003, pp. 116-144 

Links to websites 

- Program in Policy Decision-making (PPD)/Canadian 
Cochrane Network and Centre (PPD/CCNC) data- 
base: http : / / www, researchtopolicy . ca/ search/reviews . aspx 

- Source of systematic reviews of studies about health sys- 
tem arrangements (benefits, harms, key features, and the 
views and experiences of stakeholders). 

- Cochrane Library's Cochrane Database of Systematic 
Reviews (CDSR) and Database of Abstracts of Reviews of 
Effects (DARE): http://thecochranelibrary.com - Source of 
systematic reviews of programmes, services and drugs 
(benefits and possibly harms) 

- Health-evidence. ca: http://health-evidence.ca - Source of 
systematic reviews of public health programmes and serv- 
ices (benefits and possibly harms). 

- Cochrane Library's Economic Evaluation Database 
(EED): http://thecochranelibrary.com - Source of eco- 
nomic evaluations. 

PubMed: http://www.ncbi.nlm. nih.gov/pubmed 

Source of 'hedges' (i.e. validated search strategies) to find 
select types of single studies (harms, key features, and the 
views and experiences stakeholders). 

- BIREME's Virtual Health Library : http : / / www. virtual 
healthlibrary.org/php/index.php?lang=en - Source for 
many research products and databases available in the 
languages spoken in the Americas (Spanish and Portu- 
guese primarily as well as English and French). 
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